Lossless data compression is widely used in data storage and communication systems to reduce the cost and/or improve speed performance. In current practice, lossless data compression is mostly implemented at either the application level (e.g., various databases) or filesystem level (e.g., a Z File System and A Hadoop Distributed File System). Nevertheless, application-level and filesystem-level data compression is subject to noticeable overhead in terms of central processing system (CPU) cycles and in-memory data movement.
Compression can also be implemented inside storage devices without consuming any CPU cycles and requiring any in-memory data movement on the host side, which is referred to as in-storage transparent compression. The use of internal address mapping makes it practically feasible for solid-state storage devices to implement in-storage transparent compression. Moreover, as magnetic recording hard disk drives start to employ shingled recording technology, which demands the use of internal address mapping as well, future hard disk drives could also readily support in-storage transparent compression.
However, being transparent to the host (in particular filesystems running on the host), in-storage compression cannot be directly leveraged by the host to store more data on a storage device, unless there is a mechanism to pass certain information (e.g., in-storage compression efficiency and runtime available storage capacity) to the host.
Unfortunately, existing storage hierarchies (including filesystem, device driver, and storage device input/output (I/O) protocol) do not directly support such information passing from storage devices to the host. This makes it impossible for the host to utilize the extra storage capacity created by in-storage transparent compression, unless the storage hierarchy is accordingly modified.
Unfortunately, adopting such a modification would be very difficult to achieve by current mainstream ecosystems. As a result, most storage devices do not implement in-storage transparent compression at all. Although few commercial solid-state storage devices implement in-storage compression, they utilize the extra storage space created by compression solely for their own internal use (e.g., to improve the efficiency of their internal garbage collection) without changing the storage capacity visible to the host.